Skip to content

Reactivate the agentic eval against the rebuilt CLI - #183

Merged
EiffL merged 4 commits into
clean_rebuildfrom
reactivate-eval
Aug 20, 2026
Merged

Reactivate the agentic eval against the rebuilt CLI#183
EiffL merged 4 commits into
clean_rebuildfrom
reactivate-eval

Conversation

@EiffL

@EiffL EiffL commented Aug 20, 2026

Copy link
Copy Markdown
Member

The eval workflow and its prompt were written for the pre-rebuild CLI — the Snakemake shim, lc run <output_id> --universe, ok/stale/missing states, the Containerfile + requirements.txt container model. With layer 8 landed, this brings the eval back in line with what lc actually is.

evals/prompt.md (rewritten)

  • lc materialize is the build verb; lc run <command> is the probe. States are current/behind/stale, lc status always exits 0, and lc materialize --check is the gate.
  • Recipes import from the project's own locked environment — packages arrive via uv add in the project root (what the denial messages recommend); uv pip reaches nothing a recipe sees.
  • A run refuses a dirty tree, so the agent is told to commit its edits with plain git add/git commit (the annex filter is transparent — nobody runs a git-annex command).
  • results/ is the engine's to write; a hand-placed file is now detected as a foreign write and remade — the old "fails verification" warning got teeth.
  • The seed spec is stated to be deliberately incomplete: the recipe commands carry no placeholders and the dependency contracts are partial, so completing the spec ({output}, {inputs.<id>}, {decisions.<id>}, and the matching inputs:/decisions: lists) is part of the task. This makes the existing seed's shape an intentional test of spec literacy rather than a leftover.
  • The recipe template grammar section is kept — verified it still matches astra's own grammar (astra guide, astra-tools 0.2.16).

.github/workflows/eval.yml

  • Check step: the old jq expression (.universes[].outputs[], "ok", "alias") matches nothing any verb emits now. Replaced with the designed gate — lc materialize --check --json | jq -e '.up_to_date and ((.current | length) > 0)' — which exits nonzero while anything is planned or failed, plus the non-emptiness guard the old check had. behind deliberately does not fail the gate.
  • Setup step: no scientific deps are pre-seeded — the project starts empty and declaring what the recipes import (uv add, spelled out in the prompt) is part of what the eval measures; the repo-venv uv pip install is dropped (it never reached recipes). Setup commits the seed — lc init never commits, and the first lc materialize would otherwise be refused on a dirty tree before the agent did anything.
  • Trigger: the dormant PR plumbing (comment step, concurrency group) gets its trigger back — pull_request: types: [labeled] gated on a run-eval label, alongside workflow_dispatch.
  • Artifacts: .snakemake/** exclusion → .lightcone/**; keeping .git via include-hidden-files is now a feature — the run records and annexed results are the provenance.
  • Install: lc and astra are installed as uv tools — uv tool install of the checkout (the commit under test, stamped into the version by hatch-vcs) links lc and the four git-annex executables through the mirrored entry points, exercising the exact mechanism a user install relies on; astra-tools is its own tool install, pinned to the branch dependency. No venv is activated, so the runner needs no system git-annex and VIRTUAL_ENV is never set.

Seed + trace prompt

  • evals/tasks/snae/astra.yaml drops container: Containerfile — the directive is ignored by recorded decision, and no Containerfile exists to point at.
  • trace-analysis-prompt.md names the sandbox instead of the container environment.

Verified locally

  • The seed validates against astra-tools 0.2.16 after the edit.
  • The full setup + check plumbing runs end-to-end against today's lc: lc init → overlay → astra universe generate -n baselineuv add → seed commit → lc status (three outputs, direct mode, landlock) → the --check gate correctly failing with all three outputs planned on the unbuilt project.

The agent step itself (Claude Code + astra plugin) is unchanged and needs a real dispatch run to shake out — suggest a workflow_dispatch on this branch once merged, or label this PR run-eval.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx

EiffL and others added 2 commits August 20, 2026 19:17
The eval workflow and its prompt still spoke the pre-rebuild language —
the Snakemake shim, `lc run <output_id> --universe`, ok/stale/missing
states, the Containerfile + requirements.txt environment. Bring both up
to what the CLI is now:

- prompt.md: `lc materialize` is the build verb and `lc run` the probe;
  states are current/behind/stale and `lc materialize --check` is the
  gate; recipes import from the project's own locked environment
  (`uv add`, never `uv pip`) under the sandbox; a run refuses a dirty
  tree, so the agent commits its edits with plain git; `results/` is
  the engine's to write, and a foreign write is detected. The seed spec
  is now stated to be deliberately incomplete — completing the recipe
  templates and dependency contracts is part of the task.
- eval.yml: the check step gates on `lc materialize --check --json`
  (plus a non-empty `.current`) instead of a jq shape no verb emits;
  setup seeds numpy/scipy/matplotlib into the project with `uv add` and
  commits the seed so the agent starts from a clean tree; the repo-venv
  `uv pip install` is dropped (recipes never saw it); the dormant PR
  plumbing gets its trigger back as a `run-eval` label; the artifact
  exclusions swap `.snakemake/` for `.lightcone/`.
- The seed astra.yaml drops its `container:` key — the directive is
  ignored by decision, and no Containerfile exists to point at.
- trace-analysis-prompt.md names the sandbox instead of the container
  environment.

Verified locally: the seed validates against astra-tools 0.2.16, and
the setup + check plumbing runs end-to-end against today's `lc` (the
gate correctly fails with all three outputs planned on the unbuilt
project).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
The setup step no longer pre-seeds numpy/scipy/matplotlib — declaring
what the recipes import is part of what the eval measures. In exchange
the prompt spells out the uv workflow: the project starts with no
dependencies, `uv add` is the one way packages reach a recipe (it
edits pyproject.toml, re-locks, and syncs in one step, and the result
is committed like any other edit), and lock-bypassing installs reach
nothing a recipe sees.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
@EiffL EiffL added the run-eval Run the agentic eval on this PR label Aug 20, 2026
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

✅ Eval

Metric Value
Outputs check success
Agent run success
Turns 40
Tool calls 37
Cost $1.19
Agent wall time 3m50s
Model claude-sonnet-5
lc status
  mode:    direct
  sandbox: landlock (fs: declared, network: allowed)

  · current  baseline/best_fit        203627d
  · current  baseline/hubble_diagram  203627d
  · current  baseline/residuals       203627d

3 current
Confusion & pain points (Claude analysis)

Confusion & pain points

  • lc run with a relative --output path crashed instead of writing where expected: the agent's ad-hoc test invocation (lc run python scripts/fit.py --output /tmp/lbfgs_test ...) failed with a FileNotFoundError because it referenced /tmp/lbfgs_test before creating the directory — the agent then had to mkdir -p it after the failed run. This is a minor scripting error on the agent's part, but it also reveals the agent reaching for an ad-hoc /tmp scratch path rather than using the spec's own decision/universe mechanism from the start.
  • No supported way to "dry-run" alternate decisions without touching the committed spec: to verify the optimizer/error_model/redshift_cut decision axes worked, the agent had to fabricate a throwaway universes/_test_alt.yaml, commit it, materialize it, then delete the universe and its results and commit again — three extra commits (eb9dd3f, three [DATALAD RUNCMD] commits, b6eaf9c) just for exploratory validation. This points to a real product gap: there's no lightweight way to preview a universe's outputs without permanently polluting git history, since every materialize commits.
  • lc materialize <universe> doesn't accept a bare universe name: lc materialize _test_alt errored with "no output matches _test_alt", forcing the agent to discover (via the error message itself) that it must enumerate every qualified output id (_test_alt/best_fit _test_alt/hubble_diagram _test_alt/residuals). This is a discoverability gap — the CLI's own error message had to teach the agent the correct invocation, which cost a round trip.
  • RO-Crate output required a full round of manual reverse-engineering of publication mechanics: the agent had to read pyproject.toml, infer that adding license = "CC-BY-4.0" would trigger crate generation, then separately notice after a "nothing to do" materialize that ro-crate-metadata.json had appeared and been auto-committed as a second, unprompted commit (d6d191f). Nothing in the CLI output announced that the license change would produce this side effect, so the agent had to infer and verify it after the fact via ls and git log.
  • Otherwise the run was largely clean: spec validation, the fit/plot recipes, and lc materialize/--check all worked on the first real attempt once the scripts were written.

Full trace: agent-trace artifact on this run.

The engine is the host's uv tool, so the eval now installs it that way:
`uv tool install` of the checkout — the commit under test, stamped into
the version by hatch-vcs — links lc and, through the mirrored entry
points, the git-annex executables, exercising the exact mechanism a
user's `uv tool install lightcone-cli` relies on. astra's executable
belongs to astra-tools and becomes its own tool install, pinned to the
branch's dependency version.

The repo venv and its activation are gone with `uv sync`; nothing sets
VIRTUAL_ENV anymore, which also removes the benign environment-mismatch
warning uv printed at every project-scoped `uv add`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
@EiffL EiffL added run-eval Run the agentic eval on this PR and removed run-eval Run the agentic eval on this PR labels Aug 20, 2026
After every output is materialized, the agent is asked to declare an
SPDX license in pyproject.toml and materialize once more — publication
intent derived exactly as layer 8 designed it, and the crate converges
on a run that remakes nothing. The check step gates on the crate file
existing, so the RO-Crate path is exercised by every eval run instead
of warning its absence into the report.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
@EiffL EiffL added run-eval Run the agentic eval on this PR and removed run-eval Run the agentic eval on this PR labels Aug 20, 2026
@EiffL
EiffL merged commit 8b4e038 into clean_rebuild Aug 20, 2026
8 of 9 checks passed
@EiffL
EiffL deleted the reactivate-eval branch August 20, 2026 19:30
EiffL added a commit that referenced this pull request Aug 21, 2026
Widens #183's label gate: the eval runs on every non-draft PR, with
`ready_for_review` beside the default types so flipping a draft to
ready triggers the run the draft guard skips. The per-PR concurrency
group already cancels superseded runs, so a push train costs one eval.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016CRERrB5zWdVXD6uPv2BRa
EiffL added a commit that referenced this pull request Aug 21, 2026
…ibility (#190)

This merges the clean rebuild of lightcone-cli onto main — the full
re-architecture from the Snakemake shim to the model where **the locked
environment is the execution environment**. The pre-rebuild codebase was
deliberately stripped and functionality re-added one layer at a time,
each layer landing with its own tests, dependencies, and (since the
rewrite) docs. 21 PRs, all individually reviewed and merged into
`clean_rebuild`; this PR is the promotion.

## What lightcone-cli is now

A project is `pyproject.toml` + `uv.lock` + `astra.yaml` — uv is the
only environment substrate, and mode is derived, never configured:
direct by default (recipes in the project's `.venv` under
Landlock/Seatbelt), containerized when `[tool.lightcone.image]` is
declared (recipes in a content-addressed image archived in the
repository itself). Five verbs: `init`, `materialize`, `status`, `run`,
`build`.

- **Identity**: `definition_version` (what an output *is*) and
`env_version` (what it ran under) are separate on purpose — a spec
contradiction is `stale` and remade; an environment move is `behind`,
reported and left alone. `--refresh` widens by exactly that one state.
- **Storage**: the DataLad model — git carries history, git-annex
carries bytes, behind ordinary `git add`/`git commit`. Every output is
committed with a manifest and a run record `datalad rerun` can replay;
runs start and end on a clean tree; foreign writes are detected by
history and classify stale.
- **Execution**: one exec boundary (`Policy` → pure argv rewrite →
honest attestation) shared by recipes and the `lc run` probe; a recipe
writes its own output directory and nothing else; denials carry
copy-pasteable remedies. Network is uncontrolled on every mechanism,
symmetrically attested.
- **Venues**: detected, never configured — a SLURM allocation is spanned
with one worker per node; known centers get a login guard whose refusal
prints the center's own `salloc`/`sbatch` spellings; podman-hpc makes
multi-node containerized runs real.
- **Publication**: declaring `[project].license` turns on the RO-Crate
view — `ro-crate-metadata.json` converged by materialize, validated
against Provenance Run Crate 0.5 in CI. Deposit is `git archive`.
- **The engine is the host's uv tool, never a project dependency** — a
project's lock carries only what the analysis imports, and an engine
upgrade rebuilds nothing.

## The layers, as merged

| PR | Layer |
|---|---|
| #173 | Layer 5: the exec boundary and `lc run` (landed first — the
smallest honest consumer of the boundary) |
| #175 | Layers 2 + 4: identity, provenance, `lc materialize` |
| #178 | The engine is the host's uv tool (deletes layer 3 by decision)
|
| #180 | Layer 6: the container hatch |
| #181 | Layer 7: venues — SLURM in-allocation, podman-hpc |
| #182 | Layer 8: the publication view |
| #184 | Hardening: the recorded residues closed (UV_* scrub,
write-scope narrowing, symmetric network attestation, machine-level
uv.toml advisory) |
| #183 | The agentic eval reactivated: `uv tool install` of the branch
under test, empty-environment task, RO-Crate gate — three green runs |
| #185#189 | The documentation rewritten end to end: README, user
guide, CLI reference, developer corner, governance — every command block
executed before shipping, every refusal quoted from a real run |

## Verification

- The suite (tests, lint, mypy strict) is green on `clean_rebuild`,
including the three gated suites CI requires: sandbox enforcement
(Landlock on Linux, Seatbelt on macOS), container smoke (real build →
archive → bytes-free-clone `datalad rerun`), and crate smoke (the
official validator, REQUIRED clean).
- The agentic eval passes end-to-end on this branch: an agent builds a
supernova-cosmology analysis from the spec, manages the environment with
`uv add`, materializes through the real engine, and publishes a crate —
$0.70–$1.22 per run.
- `zensical build` renders the rewritten docs with zero issues; the site
deploys on the next release.

## Known deferred work (tracked, not blocking)

- The one-time **Perlmutter spike** (layer 7's pending items are listed
in CLAUDE.md) — HPC support ships honestly labeled "early days".
- **hub/GKE + Cloud Build** venue and **apptainer/singularity** runtimes
— design headroom recorded, deliberately not stubbed.
- Multi-arch image archives; the design-doc drop and CLAUDE.md slimming
when the rebuild formally closes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-eval Run the agentic eval on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant